Information processing method, terminal apparatus and information processing system

ABSTRACT

An information processing method executed by a processor included in an information processing apparatus that includes a display device that displays an image and a speaker, the information processing method includes determining, when update of an image displayed on the display device is detected while the information processing apparatus is in a state of voice call, notification sound corresponding to a substance of the update; and outputting the notification sound from the speaker.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority fromthe prior Japanese Patent Application No. 2014-128486 filed on Jun. 23,2014, the entire contents of which are incorporated herein by reference.

FIELD

The embodiments discussed herein are related to a telephone terminalapparatus, an information processing method and an informationprocessing system.

BACKGROUND

A user can operate a touch panel display device or the like to input anoperation signal to an information processing unit such as a processorcircuit in a portable telephone terminal apparatus such as a smartphone.The information processing unit performs data processing in response tothe operation signal to create given image data. Then, the informationprocessing unit causes a display device of the portable telephoneterminal apparatus to display an image of the image data.

Incidentally, while the user looks at an image displayed on the displaydevice, the user sometimes uses the telephone function and applicationsoftware for voice call of the portable telephone terminal apparatus(the telephone function and the application software for voice call arehereinafter referred to collectively as voice call function, and theapplication software is hereinafter referred to as application). Thisis, for example, a case in which, in a state in which an image by aparticular application is displayed on the display device, a callrequest is received from a different user or a like case. In order tomake it possible for the user to use the voice call function, theportable telephone terminal apparatus includes a speaker and amicrophone. It is assumed that, in such a situation as just described,the voice call function is started up while the particular applicationis in an activated state, and after the voice call is started, while thevoice call function operates on the background, the particularapplication that displays an image operates on the foreground. In thiscase, in order to listen to voice call sound of the different userissued from the speaker, the user of the portable telephone terminalapparatus would hold the portable telephone terminal apparatus in theproximity of an ear of the user and it is difficult to visually observethe display substance of the display device. Therefore, even if thesubstance displayed as a result of processing of the particularapplication changes during the voice call, it may be difficult for theuser to recognize the change. As a related art, for example, JapaneseLaid-open Patent Publication No. 2008-167373 is disclosed.

In a telephone terminal apparatus including an image display device,within a period within which a user talks using the voice call functionthe telephone terminal apparatus includes, it may be difficult for theuser to visually observe the image display device and to recognize anupdate of an image occurring within the period. A similar problem occursalso when the user holds the telephone terminal apparatus in theproximity of an ear in order to listen to sound of voice, music or thelike reproduced from a speaker in addition to the voice call sound.

SUMMARY

According to an aspect of the invention, an information processingmethod executed by a processor included in an information processingapparatus that includes a display device that displays an image and aspeaker, the information processing method includes determining, whenupdate of an image displayed on the display device is detected while theinformation processing apparatus is in a state of voice call,notification sound corresponding to a substance of the update; andoutputting the notification sound from the speaker.

The object and advantages of the invention will be realized and attainedby means of the elements and combinations particularly pointed out inthe claims.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory and arenot restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram depicting a general configuration of a telephoneterminal apparatus;

FIG. 2 is a block diagram depicting a hardware configuration of atelephone terminal apparatus according to a first embodiment;

FIG. 3 is a functional block diagram of a processor in the firstembodiment;

FIG. 4 is a processing flow chart of a processor when an operationsignal is received in the first embodiment;

FIG. 5 is a processing flow chart of a processor when a detection signalis received in the first embodiment;

FIG. 6A is a diagram illustrating an estimation method of an update typeof an image in the first embodiment;

FIG. 6B is another diagram illustrating an estimation method of anupdate type of an image in the first embodiment;

FIG. 6C is a further diagram illustrating an estimation method of anupdate type of an image in the first embodiment;

FIG. 7 is a diagram illustrating an estimation method of an update typeof an image in the first embodiment;

FIG. 8 is a processing flow chart of a processor relating to differenceextraction and generation of notification sound data in the firstembodiment;

FIG. 9 is a block diagram depicting a configuration of a thin clientsystem according to a second embodiment;

FIG. 10 is a block diagram depicting a hardware configuration of aninformation processing apparatus in the second embodiment;

FIG. 11 is a functional block diagram of a processor of a telephoneterminal apparatus in the second embodiment;

FIG. 12 is a functional block diagram of a processor of an informationprocessing apparatus in the second embodiment;

FIG. 13 is a processing flow chart of a processor of an informationprocessing apparatus in the second embodiment when an operation signalis received;

FIG. 14 is a processing flow chart of a processor of an informationprocessing apparatus in the second embodiment when a detection signal isreceived;

FIG. 15 is a block diagram depicting a configuration of a thin clientsystem according to a third embodiment;

FIG. 16 is a block diagram depicting a hardware configuration of aterminal apparatus in the third embodiment; and

FIG. 17 is a diagram depicting an estimation method of an update type ofan image in the third embodiment.

DESCRIPTION OF EMBODIMENTS First Embodiment

In a first embodiment, a telephone terminal apparatus including an imagedisplay device such as a smartphone is provided with a detection unitconfigured to detect that the telephone terminal apparatus is in use forvoice call. If the detection unit detects that the telephone terminalapparatus is in use for voice call, namely, in a state in which thetelephone terminal apparatus is in use as a voice call tool, it issupposed that it may be difficult for the user to look at the imagedisplay device because the user holds the telephone terminal apparatusin the proximity of an ear of the user. In such a case as justdescribed, when the substance of an image displayed on the image displaydevice is updated, the telephone terminal apparatus notifies the user ofa manner of the update of the substance of the image by sound.Consequently, even when it is difficult for the user to confirm thedisplay substance on the image display device by visual observation, theuser can recognize from the sound notification that the image has beenupdated. For example, if a notification that a mail is received isdisplayed on the screen while the user is talking using the telephoneterminal apparatus, then notification sound corresponding to theupdating of the screen display is issued from a speaker of the telephoneterminal apparatus.

In the present specification, the term “image” is used to include notonly a moving picture or a still picture visually representing an eventbut also an image of a character or characters displayed as text data onthe display device. In the present specification, the term “sound” isused to include voice, imitative sound and machine sound.

FIG. 1 is a diagram depicting a general configuration of a telephoneterminal apparatus such as a smartphone. Referring to FIG. 1, atelephone terminal apparatus 100 includes an image display unit 165 thatis a display device provided on one face of a housing and configured todisplay an image thereon, and a sound outputting unit 155 in the form ofa speaker that outputs voice call sound and so forth. The telephoneterminal apparatus 100 can include, as an operation inputting apparatusthat accepts an input from a user, a keyboard of a touch panel type inthe image display unit 165. Alternatively, the telephone terminalapparatus 100 may include a physical keyboard mounted at some locationof the telephone terminal apparatus 100 separately from the imagedisplay unit 165. As the telephone terminal apparatus 100, a portabletelephone terminal apparatus other than a smartphone, a personalcomputer (PC) or a personal digital assistant (PDA) of a touch paneltype in which a voice call function is provided additionally or a likeapparatus may be used.

FIG. 2 is a block diagram depicting a hardware configuration of atelephone terminal apparatus. The telephone terminal apparatus depictedin FIG. 2 may be the telephone terminal apparatus 100 depicted inFIG. 1. The telephone terminal apparatus 100 includes a processor 110, avolatile memory 130, a nonvolatile memory 135, a communication module140, an amplification circuit 150, the sound outputting unit 155, and asound inputting unit 156. The telephone terminal apparatus 100 furtherincludes a driver circuit 160, the image display unit 165, a proximitysensor 170, an operation inputting apparatus 175, a battery 180, a powersupply controlling circuit 185 and a bus 190. The processor 110 performscontrol of the entire telephone terminal apparatus 100, a generationprocess of image data and sound data and so forth. The processor 110 isan electronic circuit part such as a central processing unit (CPU), amicro-processing unit (MPU), a digital signal processor (DSP), afield-programmable gate array (FPGA) or the like. The volatile memory130 stores data and so forth that allow the processor 110 to perform agiven process. The volatile memory 130 may function as a frame bufferthat stores image data produced by the processor 110. The volatilememory 130 is an electronic circuit part such as a dynamic random accessmemory (DRAM), a static random access memory (SRAM) or the like. Thenonvolatile memory 135 stores a computer program and so forth to beexecuted by the processor 110. The nonvolatile memory 135 is anelectronic circuit part such as a mask read only memory (mask ROM), aprogrammable ROM (PROM) or a flash memory.

The communication module 140 performs data communication of sound dataand image data with a different information processing apparatus such asa terminal apparatus or a server. The data communication may beperformed by any of wire communication and wireless communication. Wherewireless communication is applied, the communication module 140includes, for example, a wireless circuit, a baseband circuit, anamplification circuit, an antenna and so forth. A communication moduleunit that implements the voice call function of the telephone terminalapparatus 100 and a communication module unit that implements differentdata communication may be provided separately as the communicationmodule 140. The amplification circuit 150 is a circuit that amplifies asignal relating to sound data and transmits the amplified signal to thesound outputting unit 155. The amplification circuit 150 is anelectronic circuit part such as a differential amplification circuit.The sound outputting unit 155 outputs sound based on a signal receivedfrom the amplification circuit 150. The sound outputting unit 155 is aspeaker provided, for example, on the front face of the telephoneterminal apparatus 100. The sound inputting unit 156 is an inputtingdevice that inputs sound when the user uses the telephone terminalapparatus 100 as a telephone tool. The sound inputting unit 156 is amicrophone provided, for example, on the front face of the telephoneterminal apparatus 100.

The driver circuit 160 is a driver circuit that outputs image datastored in the volatile memory 130 and so forth to the image display unit165. The image display unit 165 is a display device that displays theimage data received from the driver circuit 160. The proximity sensor170 is a sensor that is provided, for example, in the proximity of thesound outputting unit 155 and detects that the telephone terminalapparatus 100 is positioned closely to an ear of the user. The proximitysensor 170 detects a material body existing at a given distance from theproximity sensor 170 and outputs a detection signal. The given distanceis, for example, 5 cm. This distance is an exemplary one and is suitablyset taking the ease of use of the telephone terminal apparatus 100 bythe user into consideration. The proximity sensor 170 is an example of amechanism that detects that the voice call function of the telephoneterminal apparatus 100 is in use by the user. As the proximity sensor170, a sensor of the capacitor type, ultrasonic type, electromagneticwave type, induction type, infrared ray type or a like type can be used.As another example of the detection mechanism, a mechanism that uses anoptical sensor or an illumination sensor, a gyro or an accelerationsensor or a like sensor to grasp a state in which the telephone terminalapparatus 100 is held or a like state.

The battery 180 is a power supply source for the telephone terminalapparatus 100. The battery 180 charges when the telephone terminalapparatus 100 is coupled to a power supply adapter. The battery 180discharges when the telephone terminal apparatus 100 is separated fromthe power supply adapter. The battery 180 supplies sufficient charge tothe circuitry in the telephone terminal apparatus 100. The power supplycontrolling circuit 185 is a control circuit for the battery 180. Thepower supply controlling circuit 185 performs, for example, control ofthe charge rate and controls the discharge rate in order to suppress thepower dissipation of the telephone terminal apparatus 100 in accordancewith an instruction from the processor 110. The power supply controllingcircuit 185 is, for example, a direct current-direct current (DC-DC)conversion circuit. The bus 190 interlinks the circuits of the processor110 and the volatile memory 130 and so forth and functions as a path fordata transmission between the circuits.

FIG. 3 is a functional block diagram of a processor. The processordepicted in FIG. 3 may be the processor 110 depicted in FIG. 2. Theprocessor 110 implements the functions depicted in FIG. 3 by executing agiven computer program stored in the nonvolatile memory 135 and soforth. The processor 110 functions as a detection signal reception unit111, an operation signal reception unit 112, a control unit 113, aresponse sound data generation unit 114, a frame data generation unit115, a frame buffer 116, and a difference extraction unit 117. Theprocessor 110 further functions as a notification sound data generationunit 118, a voice call sound signal reception unit 119, a voice callsound signal decoding unit 120 and a sound data synthesis unit 121. Thedetection signal reception unit 111 receives a detection signal from theproximity sensor 170 when the proximity sensor 170 detects a target. Theoperation signal reception unit 112 receives an operation signal whenthe user operates the operation inputting apparatus 175 to performinputting to the telephone terminal apparatus 100. The operation signalreception unit 112 receives, for example, data inputted by the userusing the operation inputting apparatus 175 such as a keyboard or anoperation signal inputted by a browsing operation by the user in orderto access a particular content of a web site being displayed.

The control unit 113 performs a given data process using the responsesound data generation unit 114 or the frame data generation unit 115hereinafter described in response to an operation signal received by theoperation signal reception unit 112. The control unit 113 generatesframe data to be displayed on the image display unit 165 in response tothe operation signal, music data of response sound to be reproduced bythe sound outputting unit 155 in response to the operation signal, andso forth. The control unit 113 is implemented, for example, from theKernel layer.

The response sound data generation unit 114 generates response sounddata in accordance with an instruction from the control unit 113 andtransmits the generated response sound data to the sound data synthesisunit 121. In the present specification, sound outputted as a response toan operation input from the user inputted to the telephone terminalapparatus 100 is hereinafter referred to as “response sound.” Then,sound received from the communication destination by the voice callfunction of the telephone terminal apparatus 100 is hereinafter referredto as “voice call sound.” Further, sound for notifying the user of amanner of image update in response to update of the substance of animage as hereinafter described is hereinafter referred to as“notification sound.”

The frame data generation unit 115 generates frame data of an image tobe displayed on the image display unit 165 in accordance with aninstruction from the control unit 113, for example, a graphics deviceinterface (GDI) command or the like. The frame data generated by theframe data generation unit 115 is stored into the frame buffer 116. Theframe data stored in the frame buffer 116 is displayed on the imagedisplay unit 165 by the driver circuit 160. The difference extractionunit 117 includes a data buffer. The difference extraction unit 117acquires frame data inputted to the frame buffer 116 (preceding framedata). Then, the difference extraction unit 117 retains the acquiredpreceding frame data in the data buffer until frame data of a next frame(next frame data) is inputted to the frame buffer 116. The differenceextraction unit 117 extracts a difference (differential) between thepreceding frame data retained therein and the next frame data inputtedto the frame buffer 116. In particular, the difference extraction unit117 extracts data corresponding to a portion at which the displaysubstance is updated between one frame and another frame displayed nextto the one frame. After extracting the difference, the differenceextraction unit 117 overwrites the preceding frame data retained thereinwith the next frame data retained in the frame buffer 116. Thenotification sound data generation unit 118 estimates based on thedifference extracted by the difference extraction unit 117 what changeis indicated by the image substance. Then, the notification sound datageneration unit 118 selects notification sound in response to a resultof the estimation. Details of a method of estimating a type of update ofthe image substance based on the difference in the frame data arehereinafter described. The notification sound data generation unit 118generates notification sound data corresponding to the selectednotification sound and transmits the notification sound data to thesound data synthesis unit 121.

When the user communicates with a different telephone terminal apparatususing the voice call function of the telephone terminal apparatus 100,the voice call sound signal reception unit 119 receives a sound signalof voice call sound received from the different telephone terminalapparatus. The voice call sound signal decoding unit 120 decodes thevoice call sound signal received by the voice call sound signalreception unit 119 to generate voice call sound data. The voice callsound signal decoding unit 120 transmits the generated voice call sounddata to the sound data synthesis unit 121.

The sound data synthesis unit 121 performs synthesis (mixing) of aplurality of pieces of sound data inputted thereto. For example, whenresponse sound data is inputted from the response sound data generationunit 114 and besides voice call sound data is inputted from the voicecall sound signal decoding unit 120, the sound data synthesis unit 121synthesizes these pieces of sound data. For example, if voice call sounddata is inputted from the voice call sound signal decoding unit 120 andbesides notification sound data is inputted from the notification sounddata generation unit 118, then the sound data synthesis unit 121synthesizes these pieces of sound data. The synthesized sound data isreproduced by the sound outputting unit 155 through the amplificationcircuit 150. In this case, the user can hear the notification sound inaddition to the voice call sound. Even during voice call in which thevoice call function is used, the user can recognize the manner of updateof the image.

All of the functional blocks depicted in FIG. 3 need not be implementedby the processor 110. For example, the frame buffer 116 may beimplemented otherwise using the volatile memory 130 that is an externaldevice to the processor 110. The voice call sound signal reception unit119 and the voice call sound signal decoding unit 120 may be implementedusing a baseband circuit of the communication module 140 or a soundcodec circuit for exclusive use. Further, the sound data synthesis unit121 may be implemented by synthesis of a plurality of analog soundsignals using the amplification circuit 150 as an adder amplifiercircuit. In particular, voltage signals obtained by analog conversion ofa plurality of pieces of sound data such as response sound data are eachinputted in parallel to the amplification circuit 150 through resistanceelements. In this case, the sound volumes of the sounds can be adjustedsuitably by controlling the resistance value of each resistance element.

FIG. 4 is a flow chart of a processing executed by a processor when anoperation signal reception unit receives an operation signal from auser. The processor and the operation signal reception unit describedwith reference to FIG. 4 may be the processor 110 depicted in FIG. 2 andthe operation signal reception unit 112 depicted in FIG. 3,respectively. Processing is started at step 1000. At step 1001, theoperation signal reception unit 112 receives an operation signal fromthe user. At step 1002, the frame data generation unit 115 generatesframe data of an image to be displayed on the image display unit 165 inresponse to the operation signal. At step 1003, the frame buffer 116stores the generated frame data. At step 1004, the frame buffer 116transmits the frame data stored therein to the driver circuit 160.

After the operation signal reception unit 112 receives the operationsignal at step 1001, the response sound data generation unit 114generates response sound data at step 1005. At step 1006, the sound datasynthesis unit 121 transmits the generated response sound data to theamplification circuit 150. After steps 1004 and 1006, the processing isended at step 1007. In the description given above with reference toFIG. 4, a case is assumed in which the processor 110 receiving anoperation signal from the user generates response sound data in additionto frame data. However, where the processor 110 generates only framedata but does not generate response sound data, execution of theprocesses at steps 1005 and 1006 is omitted.

FIG. 5 is a flow chart of processing executed by a processor when adetection signal reception unit receives a detection signal from acommunication state detection mechanism such as a proximity sensor. Theprocessor, the detection signal reception unit and the proximity sensordescribed with reference to FIG. 5 may be the processor 110 depicted inFIG. 2, the detection signal reception unit 111 depicted in FIG. 3 andthe proximity sensor 170 depicted in FIG. 3, respectively. Processing isstarted at step 1100, and the detection signal reception unit 111receives a detection signal at step 1101. At step 1102, the differenceextraction unit 117 extracts, based on frame data corresponding to atleast two frames stored in the frame buffer 116, a difference betweenthe pieces of frame data. At step 1103, the notification sound datageneration unit 118 estimates a type of update of the image based on thedifference between the pieces of frame data extracted by the differenceextraction unit 117. As an example of the type of update of the image,that a notification display image that a mail is received is displayedby popup display on the image display unit 165 or that a moving pictureis reproduced to successively update the image is available.

At step 1104, the notification sound data generation unit 118 selectsnotification sound based on the estimated type of update of the image.Then, the notification sound data generation unit 118 generatesnotification sound data corresponding to the selected notificationsound. At step 1105, the sound data synthesis unit 121 synthesizes thenotification sound data generated by the notification sound datageneration unit 118 with the response sound data generated by theresponse sound data generation unit 114 or the voice call sound datagenerated by the voice call sound signal decoding unit 120. At step1106, the sound data synthesis unit 121 transmits the synthesized sounddata to the amplification circuit 150. When none of response sound dataand communication sound data is generated, the sound data synthesis unit121 transmits only the notification sound data to the amplificationcircuit 150. After step 1101, the control unit 113 turns off the imagedisplay unit 165 at step 1107. Consequently, for a period of time withinwhich the user continues to talk using the telephone terminal apparatus100, power to be used for display of the image display unit 165 can bereduced. As a method for turning off the image display unit 165, amethod of stopping the power supply to the image display unit 165 or,where the image display unit 165 is a liquid crystal display device, amethod of reducing the illuminance of or turning off the backlight maybe applied. In addition to the turning off the image display unit 165,the processing by the driver circuit 160 may be stopped. Step 1107 isnot an essentially required process in performing the method of thepresent embodiment but is suitably performed as one of countermeasuresfor power management of the telephone terminal apparatus 100. However,even where the power supply to the image display unit 165 or the drivercircuit 160 is turned off, processing relating to image data such as,for example, processing for generation of frame data or for extractionof a difference between pieces of frame data is executed continuously.This is because it is made possible by the processes for thenotification sound data generation unit 118 to recognize that updatingof an image has occurred and notify the user of such updating.

The synthesis of a plurality of pieces of data performed by the sounddata synthesis unit 121 may be performed such that sounds of the sounddata are reproduced simultaneously in an overlapping relationship witheach other or may be performed in a time division fashion such that thesounds are reproduced alternately. The sound data synthesis unit 121 mayperform weighting in sound volume for each piece of sound data so thatthe pieces of sound data may be outputted in individually suitable soundvolumes. For example, the sound volumes may be controlled so that thesound volume of the response sound or the notification sound is smallerthan the sound volume of the voice call sound in order that thecommunication sound can be heard readily.

FIGS. 6A, 6B and 6C are diagrams illustrating a method for selecting atype of update of an image based on frame data stored in a frame buffer.The frame buffer described with reference to FIGS. 6A, 6B and 6C may bethe frame buffer 116 depicted in FIG. 3. Extraction of difference databetween frames by the difference extraction unit 117 is performed usinga map in which the frames are divided into a plurality of unit regionsdisposed in a mesh form. FIG. 6A illustrates a manner in which a frameis divided into a plurality of unit regions disposed in a mesh form.Although a frame is divided into unit regions, for example, of 16 rowsand 16 columns, part of the unit regions, namely, a unit region group of6 rows and 9 columns, is depicted in FIG. 6A. Each unit region includesa plurality of picture elements (pixels). One unit region includespixels disposed, for example, in 8 rows and 8 columns. Reference symbol1 a in FIG. 6A denotes a unit region; 1 b a pixel; and 1 c an updaterectangular region. In FIG. 6A, a plurality of rectangular regionsindicated by slanting lines denote regions in which the displaysubstance differs between two frames displayed successively. In thepresent specification, each of the rectangular regions is referred to asupdate rectangular region. That a region in which the display substanceis updated has a rectangular shape is not the essence of the presentembodiment, and the update region of the display substance may have ashape other than the rectangular shape.

The difference extraction unit 117 counts, for each unit region, thenumber of update rectangular regions the unit region overlaps with. FIG.6B depicts a count number for each unit region in the example depictedin FIG. 6A. In any unit region that does not overlap with any of theupdate rectangular regions, the count number is “0.” The count number ina unit region that overlaps with a plurality of, for example, two,update rectangular regions is “2.”

Then, an action of counting the number of overlaps with updaterectangular regions for each unit region is executed for a fixed periodof time, for example, for one second. Then, if frame update is performedby a plural number of times during the execution of the action, then thecounted numbers are cumulatively added. Where 30 frames are displayedfor one second, counting results for 30 times of counting arecumulatively added. FIG. 6C exemplifies the count number for each unitregion cumulatively added for a fixed period of time.

Now, a method of estimating a type of update of an image to be performedby a notification sound data generation unit depicted in FIG. 7 isdescribed based on the results depicted in FIG. 6C. The notificationsound data generation unit described with reference to FIG. 7 may be thenotification sound data generation unit 118 depicted in FIG. 3. FIG. 7is a table exemplifying a relationship among an accumulated count numberfor each unit region, the number of unit regions in which theaccumulated count number is equal to or greater than a given value, thesubstance of estimation of a type and a type of notification sound. Inan example indicated in the first column of the table, where the numberof unit regions that satisfy a condition that the accumulated countnumber is “equal to or greater than 3” is “equal to or greater than 25and equal to or smaller than 50,” it is determined that the type ofupdate of the image is, for example, a mail reception notificationmessage displayed in popup on the image display unit 165 based onreception of a mail. This is based on a decision that the range in whichthe data substance is updated is limited to a fixed region of the framedata. In the example depicted in FIG. 6C, since 28 unit regions indicatean accumulated count number equal to or greater than “3,” the conditiondescribed above is satisfied. Thus, as the notification sound, machinesound of “ping” is selected.

In an example depicted in the second column of the table in FIG. 7,where the number of unit regions that satisfy a condition that theaccumulated count number is “equal to or greater than 10” is “equal toor greater than 200,” it is determined that the type of update of theimage is, for example, update of successive frames based on reproductionof a moving picture. This is based on a decision that the range of theframe data in which the data substance is updated is comparatively wideand also the number of updates is comparatively great. In this case,machine sound of “clack clack” depicting a manner in which a projectoroperates is selected as the notification sound. All notification soundsdepicted in FIG. 7 are exemplary, but some other sounds may be usedinstead.

In this manner, by determining an estimation type in advance using therelation between an accumulated count number and a unit region number,an update type of an image can be estimated based on data relating todifferences acquired by the difference extraction unit 117.

FIG. 8 is a flow chart of processing relating to extraction of adifference between pieces of frame data and generation of notificationsound data. Processing is started at step 1200. At step 1201, the framebuffer 116 receives new frame data. At step 1202, the differenceextraction unit 117 extracts a difference between the frame data in thepreceding cycle and the newly received frame data and determines whetheror not an update rectangular region exists. If it is determined at step1202 that an update rectangular region does not exist, then theprocessing returns to step 1201. Then, the processing is waited untilnext new frame data is stored into the frame buffer 116. If it isdetermined at step 1202 that an update rectangular region exists, thenthe processing advances to step 1203. At step 1203, in regard to a unitregion that includes an overlap with an update rectangular region orupdate rectangular regions, the difference extraction unit 117 countsand adds the number of update rectangular regions including an overlapwith the unit region. At step 1204, the difference extraction unit 117determines whether or not a given period of time, for example onesecond, has elapsed. If it is determined at step 1204 that the givenperiod of time has not elapsed, then the processing returns to step1201. Then, steps 1201, 1202 and 1203 are repeated until the givenperiod of time elapses. This given period of time is set suitably takingthe accuracy in estimating the type of update of an image intoconsideration. For example, where the given period of time is short, theaccumulated amount of data to be used for estimation is small, andtherefore, it may be difficult to suitably decide the type of update. Ifthe given period of time is long, image updates of a plurality ofdifferent types may occur within the given period of time, andtherefore, it may be difficult to select suitable notification sound.Accordingly, taking the accuracy in estimating the type of update of animage into consideration, the given period of time is set, for example,to a “period of time in which data for 30 frames is to be acquired,” orto “one second” or the like.

If it is determined at step 1204 that the given period of time haselapsed, then the processing advances to step 1205. At step 1205, thenotification sound data generation unit 118 estimates the type of updateof the image based on the result of the counting by the differenceextraction unit 117. In particular, such a decision table as depicted inFIG. 7 is stored in advance, for example, in the volatile memory 130.Then, the notification sound data generation unit 118 refers to thedecision table based on the counting result obtained at step 1203 tospecify the type of update. At step 1206, the difference extraction unit117 selects notification sound corresponding to the estimated type ofupdate. At step 1207, the notification sound data generation unit 118generates notification sound data corresponding to the selectednotification sound and transmits the notification sound data to thesound data synthesis unit 121. Then, the processing ends at step 1208.

In this manner, in the first embodiment, when the user of the telephoneterminal apparatus 100 uses the voice call function of the telephoneterminal apparatus 100 and holds the telephone terminal apparatus 100 inthe proximity of an ear, notification sound corresponding to the type ofupdate based on update of the substance of the image displayed on theimage display unit 165 is reproduced from the sound outputting unit 155of the telephone terminal apparatus 100. Consequently, even when it isdifficult for the user to visually observe the image display unit 165,the user can recognize, from the sound, update of the image displayed onthe image display unit 165. The user who recognizes from thenotification sound that the image substance has been updated canvisually observe and recognize the display substance of the imagedisplay unit 165 with the telephone terminal apparatus 100 moved awayfrom the ear once. Thus, the convenience of the telephone terminalapparatus 100 can be improved. The technology disclosed through thepresent embodiment exhibits its effects also in other cases than thecase in which the user uses the voice call function of the telephoneterminal apparatus 100. For example, also where the user holds thetelephone terminal apparatus 100 in the proximity of an ear in order tohear music reproduced, for example, from the sound outputting unit 155or sound other than voice call sound such as, for example, responsesound, the user is notified of update of the image displayed on theimage display unit 165 by sound.

Here, types of notification sound are additionally described. Thenotification sound may be any of machine sound and voice of a humanspeech. For example, where a display image for the notification ofreception of a mail is popped up, the machine sound of “ping” may beused as the notification sound, or a voice “a mail has been received”may be used as the notification sound. For example, as the notificationsound when a moving picture is reproduced, machine sound of “clack clackclack” generated when a projector operates may be used. Alternatively, avoice “a moving picture is being reproduced” may be used as thenotification sound.

As the method of recognizing that an image is updated, the method ofextracting a difference between pieces of frame data stored in the framebuffer 116 has been presented as an example. However, some other methodmay be used for recognizing update of an image. For example, a method ofextracting a difference between contents such as text data to bedisplayed on the image display unit 165 so that it is estimated that animage to be displayed on the image display unit 165 is updated may beused.

Second Embodiment

In a second embodiment, a thin client system that includes a telephoneterminal apparatus and a server that receives an operation signal fromthe telephone terminal apparatus and transmits given image data or sounddata to the telephone terminal apparatus is taken as an example, and atechnique for notifying a user of the telephone terminal apparatus of atype of update of an image is described. FIG. 9 is a block diagramdepicting a configuration of a thin client system. The telephoneterminal apparatus depicted in FIG. 9 may be the telephone terminalapparatus 100 depicted in FIG. 1. As depicted in FIG. 9, the telephoneterminal apparatus 100 is coupled to an information processing apparatus300 such as a sever through a network 500. As the network 500, acommunication network of an arbitrary type such as the Internet, a localarea network (LAN) or a virtual private network (VPN) can be adoptedirrespective of whether it is of the wire type or the wireless type.

The information processing apparatus 300 includes a function ofproviding a service for remotely controlling image data to be displayedon the telephone terminal apparatus 100 in response to an access theretofrom the telephone terminal apparatus 100. In the thin client system, ifan operation input from a user is performed for the telephone terminalapparatus 100, then the operation input is transmitted to theinformation processing apparatus 300 through the network 500. Then, theinformation processing apparatus 300 executes actual data processing inaccordance with the operation input of the user. Then, the informationprocessing apparatus 300 transmits a result of the processing and dataheld by the information processing apparatus 300 to the telephoneterminal apparatus 100 through the network 500. The telephone terminalapparatus 100 causes the processing result and the data transmittedthereto from the information processing apparatus 300 to be displayed onthe image display unit 165. Consequently, the user can use the telephoneterminal apparatus 100 as if the telephone terminal apparatus 100 mainlyexecuted data processing or retained data. As an example, the thinclient system can cause the information processing apparatus 300 toexecute an application relating to a work such as material preparationor a mail and cause the telephone terminal apparatus 100 to display aresult of the processing of the application.

The second embodiment discloses a technology for notifying, when a useruses the voice call function in a state in which image data distributedfrom the information processing apparatus 300 is displayed on thetelephone terminal apparatus 100 in the thin client system, the user bysound that the image substance has been updated. For example, a case isassumed that, in a state in which image data of given educationalmaterial is distributed from the information processing apparatus 300 tothe telephone terminal apparatus 100 in an e-learning system, a user whois performing e-learning uses the voice call function to perform voicecall. In this case, if a page of the educational material provided onthe e-learning system is updated, then sound for the notification that apage has been updated is outputted from the sound outputting unit 155 ofthe telephone terminal apparatus 100.

FIG. 10 is a block diagram depicting a hardware configuration of aninformation processing apparatus. The information processing apparatusdepicted in FIG. 10 may be the information processing apparatus 300depicted in FIG. 9. The information processing apparatus 300 includes aprocessor 310, a volatile memory 330, a nonvolatile memory 335, acommunication module 340, and a bus 390. The processor 310 performscontrol of the entire information processing apparatus 300 and ageneration process of image data or a generation process of sound databased on an operation signal accepted from the telephone terminalapparatus 100 through the network 500. The processor 310 is anelectronic circuit part such as a CPU, an MPU, a DSP or an FPGA. Thevolatile memory 330 stores data to be used by the processor 310 toperform processing. The volatile memory 330 may be used as a framebuffer configured to store display data produced by the processor 310.The volatile memory 330 is an electronic circuit part such as a DRAM oran SRAM. The nonvolatile memory 335 stores a computer program to beexecuted by the processor 310 and so forth. The nonvolatile memory 335is a hard disc drive (HDD) or an electronic circuit part such as a maskROM, a PROM or a flash memory. The communication module 340 controlsdata communication between the information processing apparatus 300 andthe telephone terminal apparatus 100. The communication module 340 is,for example, a network interface card (NIC). The bus 390 interlinks theprocessor 310, volatile memory 330, nonvolatile memory 335 andcommunication module 340 and functions as a communication path forperforming transmission and reception of data to and from one another.

FIG. 11 is a functional block diagram of a processor of a telephoneterminal apparatus according to the second embodiment. The telephoneterminal apparatus described with reference to FIG. 11 may be thetelephone terminal apparatus 100 depicted in FIG. 9. The processordepicted in FIG. 11 may be the processor 110 depicted in FIG. 2 with theexception described below. The processor 110 in the second embodimentexecutes a given computer program stored in the nonvolatile memory 135or the like to implement the functions depicted in FIG. 11. Thefunctional blocks including same contents as the contents of thefunctional blocks described hereinabove with reference to FIG. 3 aredenoted same reference symbols in FIG. 11 and overlapping description isomitted herein suitably. The processor 110 in the second embodimentfunctions as the detection signal reception unit 111, the operationsignal reception unit 112, the frame buffer 116, the voice call soundsignal reception unit 119, the voice call sound signal decoding unit120, the sound data synthesis unit 121, a distribution sound signaldecoding unit 122 and a frame signal decoding unit 123. The detectionsignal reception unit 111 receives a detection signal from the proximitysensor 170. In the present embodiment, the detection signal received bythe detection signal reception unit 111 is transferred to theinformation processing apparatus 300 through the communication module140. The operation signal reception unit 112 receives an operationsignal inputted to the telephone terminal apparatus 100 by the user. Inthe present embodiment, the operation signal received by the operationsignal reception unit 112 is transferred to the information processingapparatus 300 through the communication module 140. This is because, inthe thin client system, at least part of actual data processing isprocessed by the information processing apparatus 300.

The distribution sound signal decoding unit 122 receives sound datadistributed from the information processing apparatus 300 in response toan operation signal transferred from the operation signal reception unit112 through the communication module 140 and decodes the received sounddata. As described later, the sound data distributed from theinformation processing apparatus 300 to the telephone terminal apparatus100 is sound data obtained by synthesis of response sound data generatedin response to an operation signal transmitted from the telephoneterminal apparatus 100 and notification sound data generated in responseto the type of update of the substance of an image. In the presentembodiment, the sound data transmitted from the information processingapparatus 300 to the telephone terminal apparatus 100 is referred to as“distribution sound data.” The frame signal decoding unit 123 receivesan image signal distributed from the information processing apparatus300 through the communication module 140 and decodes the received imagesignal. The decoded image data is stored into the frame buffer 116 anddisplayed on the image display unit 165.

FIG. 12 is a functional block diagram of a processor of an informationprocessing apparatus. The processor and the information processingapparatus described with reference to FIG. 12 may be the processor 310and the information processing apparatus 300 depicted in FIG. 10,respectively. The processor 310 executes a given computer program storedin the nonvolatile memory 335 or the like to implement functionsdepicted in FIG. 12. The processor 310 functions as a detection signalreception unit 311, an operation signal reception unit 312, a controlunit 313, a response sound data generation unit 314, a frame datageneration unit 315, and a frame buffer 316. The processor 310 furtherfunctions as a difference extraction unit 317, a notification sound datageneration unit 318, a sound data synthesis unit 321, an image datacompression unit 324 and an information addition unit 325. The detectionsignal reception unit 311 receives a detection signal transferred fromthe detection signal reception unit 111 of the telephone terminalapparatus 100 and used for the notification that the proximity sensor170 detects a target through the network 500. The operation signalreception unit 312 receives an operation signal transferred from theoperation signal reception unit 112 of the telephone terminal apparatus100 and inputted to the telephone terminal apparatus 100 through thenetwork 500.

The control unit 313 controls the response sound data generation unit314 or the frame data generation unit 315 hereinafter described based onan operation signal received by the operation signal reception unit 312.Then, the control unit 313 generates, as a response to the operationsignal, image data, sound data or the like to be displayed or reproducedby the telephone terminal apparatus 100. The control unit 313 determinesbased on reception of a detection signal that the telephone terminalapparatus 100 is held in the proximity of an ear of the user. Then, thecontrol unit 313 performs control of the difference extraction unit 317or the notification sound data generation unit 318 hereinafterdescribed. The response sound data generation unit 314 generates soundresponse data to the telephone terminal apparatus 100, for example,sound data of the pulse code modulation (PCM) format, in accordance withan instruction from the control unit 313, and transmits the generatedsound data to the sound data synthesis unit 321.

The frame data generation unit 315 generates frame data of an image tobe displayed on the telephone terminal apparatus 100 in accordance withan instruction from the control unit 313. The frame data generated bythe frame data generation unit 315 is stored into the frame buffer 316.The difference extraction unit 317 includes a data buffer. Thedifference extraction unit 317 acquires frame data (preceding framedata) inputted to the frame buffer 316. Then, the difference extractionunit 317 retains the acquired preceding frame data in the data bufferuntil frame data of a next frame (next frame data) is inputted to theframe buffer 316. The difference extraction unit 317 extracts adifference (differential) between the preceding frame data retainedtherein and the next frame data inputted to the frame buffer 316. Inother words, the difference extraction unit 317 extracts datacorresponding to a portion at which update of the display substance isfound between one frame and a different frame that is to be displayednext to the one frame. The notification sound data generation unit 318estimates the type of update of the image substance based on thedifference extracted by the difference extraction unit 317. Then, thenotification sound data generation unit 318 selects notification soundin response to a result of the estimation. The notification sound datageneration unit 318 transmits the notification sound data correspondingto the selected notification sound to the sound data synthesis unit 321.The sound data synthesis unit 321 synthesizes the response sound datagenerated by the response sound data generation unit 314 and thenotification sound data generated by the notification sound datageneration unit 318. Then, the sound data synthesis unit 321 generatesdistribution sound data to be transmitted to the telephone terminalapparatus 100 by compression by a method such as, for example, movingpicture experts group (MPEG)-1 audio layer-3 (MP3) or Ogg Vorbis(registered trademark).

The image data compression unit 324 compresses the difference dataextracted by the difference extraction unit 317 using an imagecompression code such as, for example, joint photographic experts group(JPEG) or portable network graphics (PNG) to generate compression imagedata to be transmitted to the telephone terminal apparatus 100. Theinformation addition unit 325 generates additional information to beused for decoding such as, for example, a frame number. Then, theinformation addition unit 325 adds the generated additional informationto the compression image data generated by the image data compressionunit 324 as a header or the like to generate a transmission packet. Theinformation addition unit 325 adds suitable additional information alsoto the distribution sound data generated by the sound data synthesisunit 321 to generate a transmission packet. It is not necessary toimplement all of the functional blocks depicted in FIG. 12 by theprocessor 310. For example, the frame buffer 316 may be implementedusing the volatile memory 330 that is an external device to theprocessor 310.

FIG. 13 is a processing flow chart of a processor when an informationprocessing apparatus receives an operation signal from a telephoneterminal apparatus. The processor, the information processing apparatusand the telephone terminal apparatus described with reference to FIG. 13may be the processor 310 depicted in FIG. 10, the information processingapparatus 300 and the telephone terminal apparatus 100 depicted in FIG.9, respectively. Processing is started at step 1300. At step 1301, theoperation signal reception unit 312 receives an operation signal fromthe telephone terminal apparatus 100 through the network 500. At step1302, the frame data generation unit 315 generates frame data to bedisplayed on the image display unit 165 of the telephone terminalapparatus 100 in response to the substance of the operation signal. Atstep 1303, the frame buffer 316 stores the generated frame data. At step1304, the difference extraction unit 317 extracts, from the frame datastored in the frame buffer 316, data of a portion at which the substanceof the frame data is different from the substance of the frame data inthe preceding cycle. At step 1305, the image data compression unit 324compresses the extracted data at the different portion to generatecompression image data. By extracting and compressing data at a portionat which the substance of the frame data is different from the substanceof the frame data in the preceding cycle in this manner, the amount ofdata to be transmitted from the information processing apparatus 300 tothe telephone terminal apparatus 100 can be reduced. At step 1306, theinformation addition unit 325 adds, to the generated compression imagedata, additional information to be used for data transmission to thetelephone terminal apparatus 100 and reproduction of the image data bythe telephone terminal apparatus 100, for example, as a header of atransmission packet. At step 1307, the information addition unit 325transmits the generated compression image data to the communicationmodule 340, and the processing is ended at step 1309. Thereafter, thecommunication module 340 transmits the generated compression image datato the telephone terminal apparatus 100 through the network 500.

Where the data generated based on an operation signal and transmitted tothe telephone terminal apparatus 100 includes sound data in addition toimage data, after step 1301, the response sound data generation unit 314generates response sound data at step 1308. Also to the response sounddata generated at step 1308, in step 1306, the information addition unit325 adds additional information to be used for data transmission to thetelephone terminal apparatus 100 and reproduction of the response sounddata by the telephone terminal apparatus 100, for example, as a headerof a transmission packet.

FIG. 14 is a processing flow chart performed by a processor when adetection signal transmitted from a telephone terminal apparatus isreceived by an information processing apparatus. The processor, thetelephone terminal apparatus and the information processing apparatusdescribed with reference to FIG. 14 may be the processor 310 depicted inFIG. 10, the telephone terminal apparatus 100 and the informationprocessing apparatus 300 depicted in FIG. 9, respectively. Processing isstarted at step 1400. At step 1401, the detection signal reception unit311 receives a detection signal transmitted thereto from the telephoneterminal apparatus 100. Consequently, the processor 310 recognizes thatthe user is using the voice call function of the telephone terminalapparatus 100. At step 1402, the difference extraction unit 317 extractsa difference between the frame data stored in the frame buffer 316 andthe frame data retained by the difference extraction unit 317. At step1403, the notification sound data generation unit 318 estimates the typeof update of the image substance based on the difference between thepieces of frame data extracted by the difference extraction unit 317. Atstep 1404, the notification sound data generation unit 318 selectsnotification sound to be used for the notification of the type of updateto the user in response to the estimated type of update of the imagesubstance to generate notification sound data. At step 1405, the sounddata synthesis unit 321 synthesizes (mixes) the response sound datagenerated by the response sound data generation unit 314 and thenotification sound data generated by the notification sound datageneration unit 318 to generate distribution sound data. At step 1406,the information addition unit 325 adds, to the generated distributionsound data, additional information to be used for the transmission tothe telephone terminal apparatus 100 and reproduction by the telephoneterminal apparatus 100 to generate a transmission packet. At step 1407,the communication module 340 transmits the generated transmission packetto the telephone terminal apparatus 100 through the network 500. Theprocessing is ended at step 1409.

At step 1408, the control unit 313 may control the image datacompression unit 324 to stop the processing thereby to stop thegeneration of compression image data. It is considered that, in a statein which a detection signal is transmitted from the telephone terminalapparatus 100, it may be difficult for the user to see the image displayunit 165. Therefore, by stopping the compression process of image dataso that transmission of image data is not performed, the transmissionload to the communication module 340 and the communication load to thenetwork 500 can be reduced. Also in the telephone terminal apparatus100, it is possible to reduce the power consumption of the telephoneterminal apparatus 100 by stopping the process relating to display.

By performing, in a state in which a detection signal is beingtransmitted from the telephone terminal apparatus 100, a process forreducing the transmission frequency of compression image data from theinformation processing apparatus 300 to the telephone terminal apparatus100, the transmission load to the communication module 340 and thecommunication load to the network 500 can be reduced. In this case, thecontrol unit 313 performs control to the communication module 340 ofreducing, based on the fact that the detection signal reception unit 311receives the detection signal, the transmission frequency of compressionimage data, for example, from 30 times per second during normaloperation to five times per second. Where update of 30 frames isperformed for one second, if the transmission frequency is reduced tofive times per second, then difference data for six frames arecompressed and transmitted by single time transmission.

The substance of the flow chart of the process relating to differenceextraction between pieces of frame data and generation of notificationsound data in the second embodiment is same as that depicted in FIG. 8,and illustration and description are omitted here. However, as regardsthe main processing constituents of the processes, it is assumed thatthe following replacement of terms is performed for the descriptiongiven with reference to FIG. 8. In particular, “frame buffer 116” isread as “frame buffer 316”; “difference extraction unit 117” as“difference extraction unit 317”; and “notification sound datageneration unit 118” as “notification sound data generation unit 318.”

In this manner, in the second embodiment, the information processingthat makes use of a thin client system is taken as an example anddescribed. Also with the second embodiment, in a state in which the useris using the voice call function of the telephone terminal apparatus100, the convenience to the user, when it is difficult for the user tosee the image display unit 165, can be improved by notifying, the userby sound that the image substance is updated similarly as in the firstembodiment.

Third Embodiment

Now, a third embodiment that assumes a scene in which a plurality ofusers perform a collaboration work using a thin client system isdescribed. FIG. 15 depicts a thin client system in which a telephoneterminal apparatus and a terminal apparatus are coupled to aninformation processing apparatus through a network. The telephoneterminal apparatus, the information processing apparatus and the networkdepicted in FIG. 15 may be the telephone terminal apparatus 100, theinformation processing apparatus 300 and the network 500 depicted inFIG. 9. To a terminal apparatus 200, a telephone terminal apparatus suchas a smartphone or some other portable telephone terminal apparatus, atablet type PC, a PDA and so forth can be applied similarly to thetelephone terminal apparatus 100. In the present embodiment, theterminal apparatus 200 need not include the voice call function.

Referring to FIG. 15, it is assumed that the telephone terminalapparatus 100 and the terminal apparatus 200 access a common source ofthe information processing apparatus 300 through the network 500. It isassumed that the telephone terminal apparatus 100 and the terminalapparatus 200 cause respective image display units to display commonimage data provided from the information processing apparatus 300. It isassumed that a user of the telephone terminal apparatus 100 and a userof the terminal apparatus 200 can individually input data, a comment orthe like to the common source of the information processing apparatus300. It is assumed that, in such a scene as just described, the user ofthe telephone terminal apparatus 100 uses the voice call function of thetelephone terminal apparatus 100 in order to ask a question or make adiscussion regarding the substance of the common source of theinformation processing apparatus 300 by voice communication with theuser of the terminal apparatus 200. Then, it is assumed that the user ofthe terminal apparatus 200 uses a keyboard or a mouse of the terminalapparatus 200 to perform inputting in order to input some information tothe common source in regard to a comment from the telephone terminalapparatus 100 during voice call with the user of the telephone terminalapparatus 100. In this case, the display substance of the common sourceof the information processing apparatus 300 is updated in response tothe input by the user of the terminal apparatus 200. However, the userof the telephone terminal apparatus 100 uses the telephone terminalapparatus 100 as a communication unit and it may be difficult for theuser of the telephone terminal apparatus 100 to see the image displayunit 165 of the telephone terminal apparatus 100. Therefore, it may bedifficult for the user of the telephone terminal apparatus 100 torecognize what information has been written into the common source fromthe user of the terminal apparatus 200 or in what manner the display ofthe common source of the information processing apparatus 300 has beenupdated with the information.

In the present embodiment, when the user of the telephone terminalapparatus 100 and the user of the terminal apparatus 200 perform acollaboration work through the information processing apparatus 300,each user can recognize an inputting situation (operation situation ofthe terminal apparatus) of the opponent so that the collaboration workcan be performed with a higher degree of smoothness. In the presentembodiment, when an inputting operation of a keyboard or a mouse isperformed for the terminal apparatus 200, an operation signal istransmitted to the information processing apparatus 300. The operationsignal includes operation information indicative of what inputtingoperation is performed by the user of the terminal apparatus 200. Theoperation information includes, for example, a keyboard input, a mousemovement, a mouse wheel turning, a mouse drag and so forth. Theinformation processing apparatus 300 selects notification sound based onthe received operation information and a difference between pieces offrame data described hereinabove in connection with the secondembodiment. For example, if there is an input from the user of theterminal apparatus 200 as a result of a keyboard operation and thedisplay image is updated, then the information processing apparatus 300selects, for example, the sound “click clack” generated when a keyboardis operated as the notification sound and transmits the selected soundto the telephone terminal apparatus 100. Consequently, the user of thetelephone terminal apparatus 100 can recognize by sound that theopponent of the collaboration work is performing some informationinputting to the common source by a keyboard operation. Therefore, theuser of the telephone terminal apparatus 100 can confirm the displayunit by visual observation as occasion commands and can thereforeperform the collaboration work with a higher degree of efficiency. Inthe present embodiment, the operation signal reception unit 312 of theinformation processing apparatus 300 records the operation signalreceived from the terminal apparatus 200 and retains the operationsignal for a fixed period of time. Then, when trying to selectnotification sound, the notification sound data generation unit 318refers not only to the difference between pieces of frame data extractedby the difference extraction unit 317 but also the operation informationreceived from the terminal apparatus 200 and selects appropriatenotification sound.

FIG. 16 is a block diagram of a hardware configuration of a terminalapparatus. The terminal apparatus depicted in FIG. 16 may be theterminal apparatus 200 depicted in FIG. 15. The terminal apparatus 200may be a telephone terminal apparatus having a configuration similar tothe configuration of the telephone terminal apparatus 100. However, itis not necessary for the terminal apparatus 200 itself to include thevoice call function, and therefore, no communication detection unit isrequired. Therefore, in the present embodiment, an example in which theterminal apparatus 200 is a PC is described. The terminal apparatus 200includes a processor 210, a volatile memory 230, a nonvolatile memory235, a communication module 240, a driver circuit 260, an image displayunit 265, an operation inputting apparatus 275 and a bus 290.

The processor 210 performs control of the entire terminal apparatus 200and processing relating to image data and sound data and so forth. Theprocessor 210 is an electronic circuit part such as a CPU, an MPU, a DSPor an FPGA. The volatile memory 230 stores data and so forth to be usedby the processor 210 to perform processing. The volatile memory 230 isan electronic circuit part such as a DRAM or an SRAM. The nonvolatilememory 235 stores a program to be executed by the processor 210 and soforth. The nonvolatile memory 235 is an electronic circuit part such asa mask ROM, a PROM or a flash memory. The communication module 240performs data communication with the telephone terminal apparatus 100 orthe information processing apparatus 300. The driver circuit 260 is adriver circuit configured to output frame data stored in the volatilememory 230 or the like to the image display unit 265. The image displayunit 265 is a display device configured to display frame data receivedfrom the driver circuit 260. The bus 290 interlinks the circuits of theprocessor 210 and the volatile memory 230 and so forth and functions asa path for data transmission between the circuits. The operationinputting apparatus 275 is an inputting apparatus such as a keyboard ora mouse.

If the user of the terminal apparatus 200 performs inputting using theoperation inputting apparatus 275 such as a keyboard or a mouse, theinput is transmitted as an operation signal to the informationprocessing apparatus 300 through the network 500. The operation signalis received by the operation signal reception unit 312. The operationsignal reception unit 312 records the operation information included inthe operation signal and retains the operation information for a fixedperiod of time. The notification sound data generation unit 318 selectsnotification sound based on the operation information recorded in theoperation signal reception unit 312 in addition to an extracteddifference between pieces of frame data stored in the frame buffer 316.

FIG. 17 is a diagram depicting an estimation method of an update type ofan image and depicting an example of conditions to be used when anotification sound data generation unit selects notification sound. Thenotification sound data generation unit described with reference to FIG.17 may be the notification sound data generation unit 318 depicted inFIG. 12. For example, the first column indicates that, if the number ofunit regions in which the accumulated count number within a given periodof time is equal to or greater than 2 is 200 or more and besides therecorded operation information represents a mouse drag operation or amouse wheel operation, then it is determined that the update of theimage substance arises from a browse operation. Then, notification soundcorresponding to the browse operation is selected. The second columnindicates that, if the number of unit regions in which the accumulatedcount number within the given period of time is equal to or greater than1 is equal to or greater than 1 and equal to or smaller than 3 andbesides the operation information represents a keyboard operation, thenit is determined that the update of the image substance is update of atext. Then, notification sound corresponding to update of a text isselected.

The substance of the flow chart of the process relating to differenceextraction between pieces of frame data and generation of notificationsound data in the third embodiment is same as the substance depicted inFIG. 8, and illustration and description are omitted here. However, asregards the main processing constituents of the processes, it is assumedthat the following replacement of terms is performed in the descriptiongiven with reference to FIG. 8. In particular, “frame buffer 116” isread as “frame buffer 316”; “difference extraction unit 117” as“difference extraction unit 317”; and “notification sound datageneration unit 118” as “notification sound data generation unit 318.”Further, as regards step 1205, the description given hereinabove withreference to FIG. 8 is read such that the notification sound datageneration unit 318 estimates the type of update of an image based on acounting result of the difference extraction unit 317 and operationinformation recorded in the operation signal reception unit 312.

In this manner, in the third embodiment, when a collaboration work isperformed by a plurality of users who utilize the thin client system, auser who is holding the telephone terminal apparatus 100 applied to anear is notified of notification sound selected based on informationrelating to update of frame data and an inputting operation of anopponent user. Consequently, the collaboration work can be performedwith a higher degree of smoothness.

While the first to third embodiments have been described, the presenttechnology is not limited to the application examples disclosed inconnection with the first to third embodiments. For example, as amechanism that notifies a user of updating of an image, an examplewherein sound corresponding to the type of update is outputted from thesound outputting unit 155 of the telephone terminal apparatus 100 isdescribed. However, the notification mechanism may be some otheralternative mechanism or may include an additional mechanism. Forexample, a vibration function of the telephone terminal apparatus 100may be used such that, when an image is updated, a vibration action isexecuted to notify the user of updating of the image. In this case, bysuitably selecting the period of time for which the telephone terminalapparatus 100 is to vibrate or by suitably selecting, when the telephoneterminal apparatus 100 is caused to vibrate intermittently, the intervalbetween the vibration actions or the like, the user is permitted todistinguish the type of update of the image.

In a collaboration work in which a thin client system is used, ifoperation information inputted from a different terminal apparatus istext data from a keyboard, then the inputted text data may be convertedinto sound and outputted from the sound outputting unit 155 of thetelephone terminal apparatus 100 in addition to or in place of the soundfor the notification of the type of update of the image. The user of thetelephone terminal apparatus 100 can thereby recognize an input of theuser of the terminal apparatus 200, who is the opponent of thecollaboration work, with a high degree of particularity and accuracy,and consequently, the collaboration work can be proceeded efficiently.In conversion of text data into sound, for example, a method can beapplied that a sound conversion application is downloaded into thetelephone terminal apparatus 100 such that the processor 110 executesthe sound conversion application in response to reception of a detectionsignal by the detection signal reception unit 111 to convert theinputted text data into sound.

All examples and conditional language recited herein are intended forpedagogical purposes to aid the reader in understanding the inventionand the concepts contributed by the inventor to furthering the art, andare to be construed as being without limitation to such specificallyrecited examples and conditions, nor does the organization of suchexamples in the specification relate to a showing of the superiority andinferiority of the invention. Although the embodiments of the presentinvention have been described in detail, it should be understood thatthe various changes, substitutions, and alterations could be made heretowithout departing from the spirit and scope of the invention.

What is claimed is:
 1. An information processing method executed by aprocessor included in an information processing apparatus that includesa display device that displays an image and a speaker, the informationprocessing method comprising: determining, when update of an imagedisplayed on the display device is detected while the informationprocessing apparatus is in a state of voice call, notification soundcorresponding to a substance of the update; and outputting thenotification sound from the speaker.
 2. The information processingmethod according to claim 1, further comprising generating frame data ofan image corresponding to the operation signal and storing the generatedframe data into a memory, every time an operation signal for operatingthe information processing apparatus is received, wherein thedetermining the notification sound includes: extracting a differencebetween two pieces of frame data, which are frame data before the updateand frame data after the update, stored in the memory; and determiningnotification sound corresponding to the extracted difference.
 3. Theinformation processing method according to claim 2, further comprising:extracting a plurality of update regions each indicative of a region inwhich a display substance is different between the two pieces of framedata; acquiring, from each of a plurality of unit regions obtained bydividing each frame data at a given distance, a count numbercorresponding to the two pieces of frame data by counting the number ofthe update regions included in the unit region; repeating the countingwithin a given period of time; and acquiring a cumulative value of thecount number for each of the plurality of unit regions by calculating atotal of the count numbers acquired by the repeating, and wherein thedetermining of the notification sound includes: specifying a type of theupdate based on the cumulative value, and selecting notification soundcorresponding to the specified type.
 4. The information processingmethod according to claim 3, wherein the acquiring of the count numberincludes acquiring the count number using a map in which the pluralityof unit regions obtained by dividing the frame data in a mesh form arearranged.
 5. The information processing method according to claim 1,further comprising controlling so that an image is not displayed on thedisplay device when it is detected that the information processingapparatus is being used as a telephone apparatus.
 6. The informationprocessing method according to claim 5, wherein the controlling includesstopping power supply to the display device.
 7. The informationprocessing method according to claim 5, wherein the controlling includesreducing an illuminance of a backlight provided in the display device.8. The information processing method according to claim 1, furthercomprising detecting that the information processing apparatus is beingused as a telephone apparatus by a proximity sensor provided on theinformation processing apparatus.
 9. The information processing methodaccording to claim 1, wherein the outputting includes outputting voicecall sound in an overlapping relationship with the notification sound.10. An information processing apparatus, comprising: a display device; aspeaker; and, a processor coupled to the display device and the speakerand configured to: determine, when update of an image displayed on thedisplay device is detected while the information processing apparatus isin a state of voice call, notification sound corresponding to asubstance of the update; and output the notification sound from thespeaker.
 11. The information processing apparatus according to claim 10,wherein the processor is further configured to: generate frame data ofan image corresponding to the operation signal and storing the generatedframe data into a memory, every time an operation signal for operatingthe information processing apparatus is received; extract a differencebetween two pieces of frame data, which are frame data before the updateand frame data after the update, stored in the memory; and determinenotification sound corresponding to the extracted difference.
 12. Theinformation processing apparatus according to claim 11, wherein theprocessor is configured to: extract a plurality of update regions eachindicative of a region in which a display substance is different betweenthe two pieces of frame data; acquire, from each of a plurality of unitregions obtained by dividing each frame data at a given distance, acount number corresponding to the two pieces of frame data by countingthe number of the update regions included in the unit region; repeat thecounting within a given period of time; acquire a cumulative value ofthe count number for each of the plurality of unit regions bycalculating a total of the count numbers acquired by repeating thecounting; specify a type of the update based on the cumulative value;and select notification sound corresponding to the specified type. 13.The information processing apparatus according to claim 12, wherein theprocessor is configured to acquire the count number using a map in whichthe plurality of unit regions obtained by dividing the frame data in amesh form are arranged.
 14. An information processing system,comprising: a first information processing apparatus including a displaydevice and a speaker; and a second information processing apparatus thattransmits an image to be displayed on the display device to the firstinformation processing apparatus, wherein the first informationprocessing apparatus is configured to: determine, when update of animage displayed on the display device is detected while the informationprocessing apparatus is in a state of voice call, notification soundcorresponding to a substance of the update; and output the notificationsound from the speaker.